Neural Networks and Rational Functions

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degreeO(poly log(1/ )) which is -close, and similarly for any rational function there exists a ReLU network of size O(poly log(1/ )) which is -close. By contrast, polynomials need degree Ω(poly(1/ )) to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions. 1. Overview Significant effort has been invested in characterizing the functions that can be efficiently approximated by neural networks. The goal of the present work is to characterize neural networks more finely by finding a class of functions which is not only well-approximated by neural networks, but also well-approximates neural networks. The function class investigated here is the class of rational functions: functions represented as the ratio of two polynomials, where the denominator is a strictly positive polynomial. For simplicity, the neural networks are taken to always use ReLU activation σr(x) := max{0, x}; for a review of neural networks and their terminology, the reader is directed to Section 1.4. For the sake of brevity, a network with ReLU activations is simply called a ReLU network. 1.1. Main results The main theorem here states that ReLU networks and rational functions approximate each other well in the sense University of Illinois, Urbana-Champaign; work completed while visiting the Simons Institute. Correspondence to: your friend . Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 by the author(s). −1.00 −0.75 −0.50 −0.25 0.00 0.25 0.50 0.75 1.00 0 1 2 3 4 spike rat poly net Figure 1. Rational, polynomial, and ReLU network fit to “spike”, a function which is 1/x along [1/4, 1] and 0 elsewhere. that -approximating one class with the other requires a representation whose size is polynomial in ln(1 / ), rather than being polynomial in 1/ . Theorem 1.1. 1. Let ∈ (0, 1] and nonnegative integer k be given. Let p : [0, 1] → [−1,+1] and q : [0, 1] → [2−k, 1] be polynomials of degree ≤ r, each with≤ smonomials. Then there exists a function f : [0, 1] → R, representable as a ReLU network of size (number of nodes) O ( k ln(1 / ) + min { srk ln(sr / ), sdk ln(dsr / ) }) ,